Be able to critically appraise a randomized controlled trial
Be able to identify important sources of bias in randomized trials
Be able to describe the methods used to reduce sources of bias in randomized trials
It is useful to take a structured approach to critical appraisal—but also take the time to think about the study and consider additional sources of bias
Intention-to-treat analyses compare participants according to how they were randomized (even if they didn’t take the treatment). It seeks to reduce bias that can occur due to differential drop-out.
Masking (or “blinding”) can occur at the level of participant, investigator, data collector, and clinician. Masking seeks to reduce biases that occur if participants/investigators know which treatment a participant is receiving
Are the results of the study valid?
the degree to which the results of the study are accurate for the participants in the trial
the degree to which the results of the study are accurate for patients receiving the treatment in routine care
Good advice for critical appraisal:
Read the study and think about it.
| Critical Appraisal | |
|---|---|
| Participants | Who was recruited to the study? What were the inclusion/exclusion criteria? Who participated? |
| Intervention | What intervention did the treatment group receive? |
| Comparator | What did the control or placebo group receive? |
| Outcome | What was the primary outcome of the trial? |
Was the assignment of patients to treatments randomized?
Were the groups similar at the start of the trial?
Aside from allocated treatment, were groups treated equally?
Were all patients who entered the trial accounted for?—and were they analysed in the groups to which they were randomized?
Were measures objective or were the patients and clinicians kept “blind” to which treatment was being received?
How large was the treatment effect? (How was the primary endpoint and important second endpoints measured?)
How precise was the estimate of the treatment effect?
| Participants | Patients within 24 hours of admission with acute coronary syndrome; average age approx. 64; multiple cardiovascular risk factors; patients with risk of bleeding excluded |
| Intervention | Clopidogrel in addition to aspirin and other aspects of standard care |
| Comparator | Aspirin and other aspects of standard care |
| Outcome | Primary endpoint: cardiovascular death, nonfatal MI or stroke; important safety endpoint: major bleed |
CURE study
After we had obtained written informed consent, patients were randomly assigned to either the clopiogrel group or the placebo group by a central, 24-hour, computerized randomization service. Permuted-block randomization, stratified according to clinical centre, was used.
The CURE Investigators (2001, 495)
Random allocation requires a random number generator
Random allocation \(\neq\) haphazard allocation, alternate allocation, etc.
The unit of random allocation should be the unit of analysis
A health district decides to assess the effectiveness of the implementation of digital hospital services.
Implementation Package A is compared to Implementation Package B.
The 14 hospitals in the district are randomized to Implementation Package A or B.
28,000 separations from the hospitals in the district are expected throughout the duration of the study.
Allocate participants within a block to ensure similar sizes in each group, e.g. ABBA, BAAB, …
Ensure even numbers of participants with a specific characteristic are randomized to each group. Participants are put in stratified groups and then randomized in blocks.
More important with smaller trials
The larger the trial, the less likely there will be clinically important differences at baseline
Crude statistical comparisons at baseline are not reliable
For the difference to be important, the covariate needs to be a possible confounder (and not adjusted for in the analysis)
CURE Study
A loading dose of clopidogrel (300mg orally) or matching placebo was administered immediately, followed by clopidogrel (75mg per day) or matching placebo for 3 to 12 months (mean duration of treatment, 9 months). Aspirin (recommended dose, 75mg to 325mg daily) was started or continued simultaneously with the study drug. Follow-up assessments occurred at discharge, at one and three months, and then every three months until the end of the study.
The CURE Investigators (2001, 495)
Treatment and follow-up needs to be the same or the differences could bring in biases in some endpoints
This can be challenging when the intervention is not a medicine (e.g. cognitive behavioural therapy, acupuncture, ‘mindfulness’, physiotherapy, …)
Haake et al. (2007): Patients with chronic low back pain were randomized to ten 30-minute sessions of acupuncture, sham acupuncture, or conventional therapy (pain medications, physiotherapy). Primary endpoint: treatment response at 6 months.
Response (linked to self-reported low back pain scores):
47.6% acupuncture
44.2% sham acupuncture
27.4% conventional therapy
Information regarding drop-outs is limited
A total of 21.1 percent of patients in the clopidogrel group discontinued the study medication permanently, as compared with 18.8 percent in the placebo group.
The CURE Investigators (2001, 498)
All analyses were based on the intention-to-treat principle…
The CURE Investigators (2001, 495)
How many participants discontinued their allocated treatment?—why?—what did they receive instead?
Important to consider flow out of and into the treatment group
How were patients that discontinued or lost to follow-up analysed?
compares treatment outcomes between participants according to treatment allocation (regardless of whether they received the treatment)
A per protocol analysis compares treatment outcomes among participants who adhered to the treatment protocol
Random allocation ensures that the treatment groups are comparable at baseline
What happens during the trial is another matter:
Drop outs: participants allocated treatment who discontinue their allocated treatment
Drop ins: participants allocated placebo or control who start taking the experimental treatment
The effects of drop out/drop in/loss to follow up is unlikely to be random
How drop out/drop in/loss to follow up is analysed will influence the estimate of treatment effect provided by the study.
Starting assumptions
Consider the following scenario
The Heart Protection Study randomized \(>\) 20,000 participants with coronary artery disease or diabetes to simvastatin 40mg daily or placebo (starting 1994)
Participants allocated simvastatin or placebo could start ‘non-study’ statin if GP thought it indicated
Primary endpoint: all-cause mortality (analysed according to intention-to-treat); 12.9% simvastatin v 14.7% placebo
Of the participants allocated simvastatin: 82% remained adherent, 3% took a non-study statin, 2% took both
Of the participants allocated placebo: by year 5, 32% took a non-study statin
Jackson et al. (2006) randomized 36,000 women (50–79 years old) to calcium and vitamin D supplementation for approx. 7 years
Primary end-point analysed according to intention-to-treat: hip fractures hazard ratio 0.88 (95% CI 0.72–1.08)
But also provide a pre-planned sensitivity analysis in the adherent population (censored 6 months after identification of nonadherence)
Approx. 60% of participants were adherent (defined as taking at least 80% of study medications)
Hip fracture rate among adherers: hazard ratio 0.71 (95% CI 0.52–0.97)
CURE Study
We undertook a randomized, double-blind, placebo-controlled trial…
The CURE Investigators (2001, 494)
All primary outcomes and life-threatening and major bleeding complications were adjudicated by persons who were unaware of the patient’s treatment-group assignments.
The CURE Investigators (2001, 495)
If participants, clinicians, people involved in collecting data and/or assessing outcomes know what treatment the participant has received it can influence how they volunteer, describe and/or interpret outcomes
Masking is especially important for subjective outcomes (Hróbjartsson and Gøtzsche 2010)
Subjective outcomes includes those that rely on participant self-report (e.g. pain, nausea)
Have a go at the knowledge-check questions on Blackboard
Haake, Michael, Han-Helge Muller, Carmen Schade-Brittinger, Heinz D Basler, Helmut Schafer, Christoph Maier, Heinz G Endres, Hans Joachim Trampisch, and Albrecht Molsberger. 2007. “German Acupuncture Trials for Chronic Low Back Pain.” Archives of Internal Medicine 167 (17): 1892–8. https://doi.org/10.1001/archinte.168.9.1011-a.
Heart Protection Study Collaborative Group. 2002. “MRC/BHF Heart Protection Study of Cholesterol Lowering with Simvastatin in 20,536 High-Risk Individuals: A Randomised Placebo-Controlled Trial.” The Lancet 360 (9326): 7–22. https://doi.org/10.1016/S0140-6736(02)09327-3.
Hróbjartsson, Asbjørn, and Peter C Gøtzsche. 2010. “Placebo Interventions for All Clinical Conditions.” Cochrane Database of Systematic Reviews (Online), no. 1: CD003974. https://doi.org/10.1002/14651858.C.
Jackson, Rebecca D, Andrea Z LaCroix, Margery Gass, Robert B Wallace, John Robbins, Cora E Lewis, Tamsen Bassford, et al. 2006. “Calcium Plus Vitamin D Supplementation and the Risk of Fractures.” New England Journal of Medicine 354 (7): 669–83. https://doi.org/10.1056/NEJMoa055218.
The CURE Investigators. 2001. “Effects of Clopidogrel in Addition to Aspirin in Patients with Acute Coronary Syndromes Without ST-Segment Elevation.” New England Journal of Medicine 345 (7): 494–502.